Learning the areas of expertise of classifiers in an ensemble

There are various machine learning algorithms for extracting patterns from data; but recently, decision combination has become popular to improve accuracy over single learner systems. The fundamental idea behind combining the decisions of an ensemble of classifiers is that different classifiers most probably misclassify different patterns and by suitably combining the decisions of complementary classifiers, accuracy can be improved. In this paper, we investigate two kinds of classifier systems which are capable of estimating how much to weight each base classifier dynamically; during the calculation of the overall output for a given test data instance: (1) In “referee-based system”, a referee is associated with each classifier which learns the area of expertise of its associated classifier and weights it accordingly. (2) However, “gating system” learns to partition the input space among all classifiers. Each referee in referee-based system learns a two-class problem (i.e., whether to use or not to use a classifier) whereas a gating system learns an L-class problem assigning the input to one of L base classifiers. Our analysis on 20 datasets from different domains and a classifier pool including 21 base learning algorithms reveals that the gating system tends to concentrate on a few of the base classifiers whereas a use of referees leads to a more balanced use of the base classifiers. Moreover, in the case of referees, it is better to use a small subset of base classifiers, instead of a single one. The study shows that, by using well-trained selection unit (referee or gating), we can get as high accuracy as using all the base classifiers (to combine their decisions) with drastic decrease in the number of base classifiers used, and also improve accuracy. The improvement is significant especially in cases when none of the base classifiers has high accuracy and it indicates that selecting classifiers appears promising as a means of solving hard learning problems.